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(54) Selection facilitation on a graphical Interface 

(57) This invention allows the user to rapidly deter- 
mine the interrelatedness of icons on graphical user 
interfaces. Upon selection of an icon, for example in a 
drag-and-drop interface, all other icons to which the 
selected icon may be related are graphically high- 
lighted. This provides a user-friendly, dynamic, visual 
catalog of interrelated icons. A scenario matrix contains 
information that defines the relations among the icons 
on the graphical interface. 
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Description 

FIELD OF THE INVENTION . . 

This invention relates to computer graphical user - s 
interfaces. More specifically, the invention relates, to a 
graphical user interface (GUI), allowing users to better - 
understand the relationship between icons displayedon 
a computer screen. ...... ; 

'.' - . - . •■ . . - 10 

BACKGROUND OF THE INVENTION 

Graphical user- interfaces (GUIs) provide ways for 
users of * computers and "other devices to effectively 
communicate with the, computer. In GUIs, available is 
applications and data sets, are often represented by 
icons consisting of small graphical representations 
which can be selected by a user and moved on the 
screen. The selection of icons often takes the place of 
typing in a command using a keyboard in-order to. initi- 20 
ate a program. Jn general, icons are tiny on-screen sym- 
bols that simplify access to a program, command, or 
data file. Icons are usually activated or selected by mov- 
ing a mouse-controlled cursor onto the icon and press- 
ing a mouse button. :, ■ -■ 25 

GUIs include graphical images on computer moni- 
tors and often consist of both icons and windows. (GUIs 
may also reside on the screens of televisions, kiosks, 
and automatic teller machines (ATMs).), A computer 
window is a portion of the graphical image that appears 30 
on the monitor and is dedicated to some specific pur- , 
pose. Windows allow the user Jo treat the, graphical 
images on the computer monitor like a desktop where 
various files can remain open simultaneously. The user 
can control the size, shape, and position of the win- , 35 
dows.-. . ■ vy : - . \- 

Although the use of GUIs with icons usually simpli- 
fies a user's interactions with a computer, GUIs are 
often tedious and frustrating to use. Icons must be 
maintained in a logical manner. It is difficult to organize 40 
windows and icons when many are similarly displayed 
at the same time on a single device:; 

In a drag-and-drop. GUI, icons are selected and 
moved to a target icon to achieve a desired effect For 
example, an icon representing a computer file stored on 4S 
disk may be dragged over an icon containing an image v 
of a printer in order to print the file^ or dragged over an . : 
icon of a trash can to delete the file. A typical user's 
screen contains many icons, and only a subset of them 
will at any one time be valid, useful targets for a selected so 
icon. For example, it would not be useful to drag the icon 
representing a data file on top of an icon whose .only 
purpose is to access an unrelated multimedia applica- 
tion. . . - . . : ■ 

r ■ . • . . 55 

OBJECTIVES OF THE INVENTION 

An objective of this inventionjs an improved system -.- 
and method for organizing and using graphical displays 



on graphical interfaces. . ; - : ■ . 

Another objective of this invention is an improved 
system and method for organizing, displaying, manag- 
ing, and selecting icons and/or windows on a computer 
graphical interface. v 

SUMMARY OF THE INVENTION 

This invention permits users to conveniently view 
and manipulate related GUI icons, particularly in a drag- 
and-drop interface but also in any presentation where 
information on the relatedness of icons and/br windows 
is desired. In a preferred embodiment, when one 
icon/window is selected and/or dragged, other related 
icons are then distinguished by one of a plurality distin- 
guishing features, such as by brightening, by rounded 
corners, or by an oval shape and/or highlighted text. 
(Note that in this disclosure, "distinguishing feature" and 
"highlight" are used interchangeably.) This distinguish- 
ing allows the user to see and understand which icons 
are related to the target icon, thus creating an interac- 
tive visual index for the user and/or guiding the user to 
specific targets (drop sites) in drag-and-drop interface. 
Icons are related when they represent data, functions, 
and/or programs that can be used together. The user 
can also specify which icons are related, e.g., those in a 
similar class like office applications, word processing, or 
drawings. Strength of the relationship can also be indi- 
cated by factors such as icons being most frequently 
used or recently used together. . ■ - ■ 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of the hardware and one 
preferred embodiment of the present invention. 

Figure 2 is a drawing of two monitors with graphical 
images/selectable items on their graphical interfaces. 

Figure 3 is a flow chart showing the steps of one 
preferred method of the present invention. 

Figure 4 is a block diagram of a data structure that 
defines the interrelatedness of icons. 

Figure 5 is a flow chart showing the steps of one 
preferred method for determining most likely pairs of 
icons to be used. 

DETAILED DESCRIPTION OF THE INVENTION 

This invention permits users to conveniently view, 
organize, use, and understand relationships between 
GUI objects including icons with static or animated 
graphics, text, multimedia presentations, and TV broad- 
casts. GUI objects could also include three dimensional 
images, for example, those used in virtual reality appli- 
cations. When one data object or program, having a dis- 
crete graphical appearance on the GUI, is selected or 
dragged, target icons are highlighted. Generally, but not 
necessarily, the highlighting Easts for a time period lim- 
ited to- the time .the original data 'object is 
selected/dragged. The highlighting is used to make the 
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user aware that a relation exists between the high- 
lighted icons. Further the highlights are used to guide 
the user toward appropriate, target icons. 

The present invention is capable of running on any 
general purpose computer, system or microprocessor s 
controlled television, including units which have the abil- ' >. 
ity to present multimedia information. One preferred 
embodiment is schematically represented in a block - 
diagram in Figure 1 . A computer system 100 comprises 
a control processing unit (CPU) 110, memory storage io 
device 120, one or more, monitors or graphical inter- 
faces 130, and selection device 140 such as a mouse or 
speech recognition system 178. In one embodiment, a 
IBM RISC SYS JEM/6000 MOO comprises a control 
processing unit (CPU) 110/ Memory, storage device is 
120, one or more monitors 130,' and a mouse 140. The 
mouse 140 may be used to select graphical images like ' 
icons 160 or windows 180. (Note that graphical images* 
are also referred to as selectable items herein.) On an 
IBM RISC System/6000 multiple monitors 130 can be 20 
controlled by multiple monitor adaptor cards 115 such > 
as the IBM .RISC, System/6000 Color Graphics ^Display ' 
Adaptor. -The computer system . 100 may also have ' < 
audio input/output capability 170. An ActionMedia II Dis- 
play Adapter 175 (described in the IBM ActionMedia II- 2s 
Technical Reference) can be used for audio/video play- v 
back 170. r 

This adaptor 1 75 may also be used fcrdisplay TV * * 
broadcasts/signals 190 and other full' motion vkJeoand 
sound audioA/isual on the monitors 130i^" "v 30 

In addition, speech recognition 178- may be pro- - • 
vided (such as the IBM VoiceType Dictation Adapter). 

In an alternative embodiment, the CPU 1 10 can be: 
connected to 112 a client/server network (or LAN 111) 
to which other target monitors 130 and/or systems 100 35 
are connected. . * . ■ . 

Systems 100 that can be used to display graphical 
images; like icons and windows, are well known, i 

GUIs can be used to control any apparatus having 
a monitor. In the field of television (TV), channel selec- 40 
tion can be affected by selecting an icon consisting of 
the animated video broadcast on that channel fre- 
quency. 

Figure 2 is a block diagram of a system 200 show- 
ing graphical images/selectable items (such as icons as 
160 or windows 180, both which may contain animated 
images) appearing on a primary monitor 230. A selecta- 
ble icon 160 or window 180 may be selected 204 and 
dragged 205 on the primary monitor 230 using a mouse 
1 40 or other selecting device 1 40. > so 

The following explains the invention using an icon, > 
although the concept applies to other graphical images - 
such as windows. Once an icon 160 is selected 204, ; - 
certain icons in the set of displayed icons 261 are high- > •* . w 
lighted 262 (by color changes or other graphical means) ss 
if they are related to the. icon 160 or can receive the . ; 
select icon ,160 in a drag and drop icon interface.. For v> 
example, icons 262.may be:the only valid "drop srtestfor?' * 
selected icon,1 60. The relation between icons is defined 



as described in Figure 4 below. TTie set of displayed 
icons 263 may reside on another monitor 240' con-' : ' 
nected to the same computer (local) or connected over 
a computer network (remote). * - : " ' 

The relation between icons may be defined by a 
user profile 271. In a preferred >rf^odiment 1 , ; this is'a 
"scenario^ matrix* ^TO-on the ffibn&ry storage device : ' 
1 20. This user profile 270, whic^ <»rt ^ist as a data f He, 
may : consist of ah array of numbers indicating which " 
icons can receive other icons in a drag-and-drojb GUI or 
which icons are related to one another in a more gen- 
eral GUI. l ' ' *' ; - /J " 

Figure 3 is a flow chart showing the steps per- 
formed by ah icon facifitatbr pfbgram r 366 executed by 
system;ib0or200f; ;^ rii0 ^ / - 

The icon facfii^orprdgrarirf 300'first queries 310 if , K 
an icon 160 is selected/ 5 for example by a mouse-or 
voice-controlled cursor. Methods for determining if an 
icon is selected are well : known. If the icon 160 is 
selected 311, the program identifies 320 an icon identi- 
fier, like ian icon index number, and locates 330 the cor- - 
responding entry, a strength value; in the scenario j 
matrix 270 (or user profile 271) which may be stored as r ' 
a file on medium 1 20. (An icon index ntimber can be any 
name that uniquely identifies the icon!) TlVe searching of 
indices in files containing arrays or recorcfe of "data is 
known to those skilled - in-' -the' art:! This file may be 
updated 1 periodically by application software or the user. 
(If an icon is added to the GUI; a new entry^may be 
added to the matrix 270.) As a result of step 330; related 
icons are then highlighted 340. In other words, high- ' 
lighted 'icons are r thosie with a relationship with the 
selected icon, where the relationship is indicated by the 
strength value corresponding to a' pair containing the 
selected icon arid the related icon. Highlighting of icons " 
may be accomplished using routines supplied by any * 
standard window software (See for example, Nye, A. 
(1988) Xlib Programming Manual. O'Reilly 1 & Associ- 
ates: Cebastopol, CA which describes color manipula- ' 
tion for the purpose of highlighting). If no icons 312 are 1 
selected, all icons are unhighlighted. ■ ' '» " ' 

Figure 4 is a block diagram of one typical preferred ' ,; v 
data structure 470 that lists alphanumeric^ indices 475, 
i 476, for each selectable itern 160 and alphanumerical 
indices 480. j 477, for each of zero or more possible tar- 
get icons (261 ,-263).^ This matrix of numbers 470 is ;: ' ^ 
called a a scenario matrix, and is used for highlighting - 
related icons when a user selects an icon. The scenario ' 
matrix may be a file on disk, or data in memory, with 
designators; like numbers, typically 490 and 491 , indi- 
cating which icons are related for the purposes of high- 
lighting relevant target icons. ■ *" ^ ; 

For example, consider a matrix A(i,D 470 where i 
476 and j 477 are numerical indices denoting various •* ; - 
icons. The scenario matrix can be a square and sym- ( ; - 
metric matrix, i.e. A(i,j) = AQ,i) , if all icons i and j are 
related to each other, for example, if i can be dragged on' - : - 
top of j, and j can be dragged on top of i with a useful 
result. More generally the matrix is not symmetric if i is 
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related to application j but the reverse is not so. As an 
example, one may transfer artwork from a drawing pro- 
gram into a word processor application, but one may not 
import a document from a word processor program into 
an artwork program. 

Figure 4 shows a binary scenario matrix 270. 470 
where highlighting (related icons) is denoted by.1 (490)., 
and non-highlighting (unrelated icons) is denoted by 0 
(491). When an iGon i 476 is selected 204 or dragged 
205, row i 477 of the scenario matrix may be thought of 
as a control vector which determines which other icons 
are highlighted. (1=highlight.cr 0=nct-highlight). There- 
fore, once the i-th icon is selected, the highlighting pro- 
gram 300 examines the i-th row of .the matrix- 
Icons represented by numerical indices 480 in j 
entries 477 in this control vector may be distinguished 
with one or more distinguishing features 262, all of 
which can be accomplished by known graphical meth- 
ods in X-windows and other windows environments by 
one skilled in the art of windows programming. These 
distinguishing features could be user selectable. Distin- 
guishing features (highlights) might include icon: bright- 
ness, outlining, font, shading, color, size, shape, and/or 
animation. For example^ using shape as a distinguish- 
ing feature, an icon shape control, known in GU! inter- 
face programs, may be employed to change the shape 
of the icon from one shape, e.g. a rectangle; to another, 
such as a rounded rectangle or an oval Using fonts as 
a distinguishing feature, a three-dimensional style font 
might be used for the titles of related icons. Also, related 
icons could be shaded (or not shaded) in the same way 
as other related icons (and the selected icon). Shading 
of icons is known in GUI -interface programming. 
Another distinguishing feature is to brighten the related 
icons, e.g., appropriate target icons (drop sites) 262. 
See routines supplied by standard window software 
(see for example, Nye, A. (1988) Xlib Programming 
Manual. O'Reilly & Associates: Cebastopoi, CA which 
describes various ways to implement distinguishing fea- 
tures.) This Xlib Programming Manual is incorporated 
by reference in its entirety. 

In an alternative preferred embodiment, the sce- 
nario matrix 470 may be< used with a range of values in 
the matrix. These values* called strength values, could 
indicate a strength of the relationship between any 
given pair of icons. For example, the higher the value, 
the more likely it is that one icon is.related to another. A 
threshold value may be set so that only those icon rela- 
tions denoted by a sufficiently high value are high- 
lighted. In other words, the strength value between a 
pair of graphical images must be above a threshold 
value before the graphical images in the pair are consid- 
ered to be related. The scenario matrix (470, 270) may 
be created manually by a user, or created by application 
software. Additionally, the values, 490. or 431 typically, in 
the matrix may be provided automatically by a program 
400 which monitors a user's past usage and increments 
values 490 and 491 in the matrix if a user is "likely" to 
use icons j 477 representing various target applications 



after using icons 476 representing* various selected 
applications. Other programs 400 could use different 
criteria to change the strength values in the matrix. 
For example, Figure 5 is flow chart indicating how 

5 the scenario matrix may evolve as program 400 moni- 
tors usage of icons. When the user selects 510 an icon, 
the program 400 checks 520 to see if the scenario 
matrix value A(i,j) (490, 491) is greater than a threshold 
value T. If yes 530 the appropriate icons are highlighted 

io 550 as in 340. If no 540, #ie icons are not highlighted. 
The index 475 of the currently selected icon i is saved 
560 in a variable called save_L When the user makes 
the next selection 570 the program 400 again checks 
580 to see if the scenario matrix value A(i,j) is greater 

is than a threshold value T. If yes 590 the appropriate 
icons are highlighted 594 as fn 340. if no 592, the icons 
are not highlighted. The index 480 of the currently 
selected icon is saved 596 in a variable called savej. In 
step 598, the value A(save_J, savej) of the scenario 

20 matrix 470 is incremented at location (save_i, savej). 
In this manner, the scenario matrix reflects "most fre- 
quently used pairs" and can therefore' highlight target 
icons 262 most likely to be used in, for example, a drag 
and drop interface. Here the strength of the relationship 

25 between any given pair of icons is determined by the 
number of times the pair of icons is used together. 

Alternatively, the values in the scenario matrix can 
be normalized by the sum of all the array elements. In 
this manner, the array value reflects the number of times 

30 a pair is selected as a percent of the total number of pair 
selections. - 

. Using similar methods, "most recently used pairs" 
may also be cataloged and highlighted. This may be 
^accomplished, for example, by periodically resetting the 

35 scenario matrix values 490 and 491 to some default and 
then letting them accumulate again as in 598. Alterna- 
tively, the scenario matrix 470 may be a three-dimen- 
sional array, whose third dimension 478 contains 
values, typically 479, indicating relationships for a 

40 period of time. For example, a 2-dimensional array, typ- 
ically 485, would exist for each of one or more given 
periods of time. In this preferred embodiment, to deter- 
mine "most recently used pairs", step 330 locates icons 
in the 2-dimensional array 485, corresponding to the 

45 most recent time period 486 in order to determine which 
icons have the strongest relation (here most recent) and 
therefore are highlighted 340. Selection of other time 
periods, or groups of time periods, can be achieved by 
specifying specific 2-dimensional arrays 485 or time 

so period ranges. 

In addition, the data files 270 containing the sce- 
nario matrix 470 can be made portable so that the entire 
scenario matrix can be moved from machine to machine 
or otherwise used to customize a machine's GUI for a 

55 particular user, or to provide a common "base" scenario 
matrix for a company or organization. The scenario 
matrix would be in the form of a data file or shared mem- 
ory file that can accessed across a LAN, WAN, or other 
network. For example, in the UNIX operating system, it 
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is standard procedure to access remote file systems via 
NFS (network file.- system) protocols. In particular, 
remote files can be .accessible via ioctIO function calls. 

Given this disclosure alternative embodiments 
would become apparent to one skilled in the art. 

• q ^ ."■..> 
Claims , . 



1. A computer system comprising: 

a central, processing unit and a memory; 

one or more, graphical .interfaces each having 
two prr more graphical :images, one of the^ 
graphical, images, being a. selected graphical 
• image and one or more of the graphical images 
t - .being a set^of displayed graphical images, one 
. . or more * of i the displayed graphical^ images 
- v being a related graphical image having a rela- 
tion with the selected graphical image; r 

: :a data;, structure residing on the computer 
memory* that defines how the selected graphi- 

. ca| images and each of the related graphical 
images are related; and ■ >>. 

a process for changing one or more distin- 
guishing features of: all the related graphical 
. images when the selected graphical image is 
. selected. i > ■ >.•■■ -..t.. . 



2. 



4. 



6. 



7. 



A computer system, as in claim 1 , where the rela- 
tion is that the related graphical images are drop 
sites for the selected graphical image. 

A computer system, as in claim 1, where therela- 
tion is that the related graphical images are in a 
same class as the selected graphical image. 

A computer system, as in claim 1, where the rela- 
tion is that one or more related graphical images 
represent a related program that can be used with a 
selected program represented by the selected 
graphical image. 

A computer system, as in claim 1, where the rela- 
tion is that one or more related graphical images 
represent a related data that can be used with a 
selected program represented by,:the selected 
graphical image. 

A computer system, as in claim 1 ..where the distin- 
guishing feature includes any. one of the following: 
brightness, outlining, jfont, shading, > color, size, 
shape, and animation. t..: ' . ^ v .* i. 

A conputer system, i: as in . -claim. where , the 
selected graphipat Image is selected.by a pointing 
device. • . / ■ ia. ■ .% *r. ; ■ - . . 



A computer system, as in claim 1, where the 
selected graphical image* is selected by 1 a speech 
* recognition system. •* ' - * : 



'lO CM). 



9. A computer system, as ifrclairti f , 1 whife'the graph; 
ical interface is a television. ' " :: ' ,c : ' 
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10.{ r A'cbmputer system ^k»nprisihg: f 3 ; - 

^ i' a central processing unitand a memory; 1 '* r 

■ -one tor more graphical iriteHaces 5 each' having 
,r two or*' J mbre graphical. ; im~ages< one '' tif u 4hV 
i .graphical Imaged v fceirig % ^elec^ ^graphicaf 

image ^d ohW or mor'eof the graphical images * 
1 i beingas^ofdisplay&lto^ 

■ . . the displayed graphic^l images being a" related • 
1 graphical image having a relation with the 

» selected graphical images; : 

r- :. " a data structure, residing oh a computer mem- 
: bry.' having one or more strength values, each 
1 >\- of the strength values indicating a 1 strength' of ■ 
't the relationship betweert ^ch ; bf one or nfore " 
: parrs ; of graphical- irif&ges^each pair including 
. - the selected graphical image and one related 
^'graphical image;^and a prcfcess for changing* 
one-or more distinguishing features of each of 
* the related^ graphical images when the selected ' 
x graphical image is selected. : 

11- A computer system; as in claim 10, where the each 
of the related graphical images is in a pair with a 
strength value above a threshold. 

12. A computer system; as in claim 10* where one or 
more of the strength values is determined by a 
number of times the graphical images in a pair are 
selected as a pair. 

13. A computer system, as in claim 12, where the 
strength values are determined for a time period. ' 

14. A computer system, as in claim 1 3, where the time : * 
period is a most recent time period. - 

15- A method of showing a relationship between icons ° 
on a graphical interface of a'computer system com- 
prising the steps of: r 

selecting a selected graphical image that is.dis- 
playisd on the graphical interface; V? v i . 

- - • i *.»:.:< 

- v determining a relation between the selected* 
graphicar image and one or more related - 

"... graphical images displayed on the graphical 
. interface, rthe ; relation defined . by a - strength ■ 
value in* : a - matrix .of strength^ values, each' 
strength value indicating the strength of the - 
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relationship between a pair of graphical 
images, each pair including the selected graph- 
ical image and related graphical image; and 

distinguishing the related graphical images by s 
changing a distinguishing characteristic of 
each of the related graphical images when the 
selected graphical image is selected. 

16. A computer system comprising: 10 

a central processing unit means and a memory 
means; * 

one or more graphical interface means for dis- 15 
playing graphical images, each graphical inter- 
face means having two or more graphical 
images, one of the graphical images being a 
selected graphical image and one or more of 
the graphical images being a set of displayed 20 
graphical images, one or more of the displayed 
graphical images being a related graphical 
image having a relation with the selected 
graphical image; 

25 

a data structure means, residing on the compu- 
ter memory means, for defining how the^ 
selected graphical image and each : of the" 
related graphical images are related; and 

• 30 

a process means for changing one or more dis- 
tinguishing features of the related graphical 
images when the selected graphical image is 
selected. 
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